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Purpose Results 


The purpose of this study is to explore a 


computerized oral reading fluency (ORF) system that (1) Comparisons of WCPM Scores (2) Comparisons of Error Rates (3) Comparisons of Timed Duration (4) Word-Level Agreement 
Mean WCPM Mean Error Rate Mean Time Took to Read Passage 


Student-level Agreement Rate 


uses speech recognition software (CORE). The 
purposes were as follows. 
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(1) Compare the mean WCPM scores across: 
a) three passage lengths (short=25 words, medium=50 


50 7 
0.09- 
words, long=85 words), and ad 
b) three scoring methods (real-time, audio recording, and method method.cat | _40- method 
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(2) Compare the error rates across the passage lengths. in ee —_  neconingt “0. —_ 
(3) Compare the timing duration of read passages between: 0.05- 
human assessors in real time as in traditional ORF, and oo. 
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(4) Analyze the agreement of word-level scores (correct or 


inco rrect) across the three scoring methods. Short ee Lone Short ieaiain cone Short kictuin Leni Short Medium igng Short Medicin ions Short Medium oie Short cist Long Short edi enig Short Kieaitin Long : j ; 3 i 5 : ji 
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Scoring Method), Interaction, and Pairwise Comparisons ASR Real Time Recording a ASR-RealTime ASR-Recording RealTime-Recording 


Sample. Students response times < 2.5 secs tuk? tence eae |. — — =e 
were removed. Students’ WWCPM scores > 1.9 PassageLength *  * * | —— — * 
times different between the three scoring Short vs. Medium ; _—— te a2 I nes 
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Medium vs. Long 
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and seven Grade 3 students were removed from Scoring Method : ) aS Se ae ee 7 
the analysis sample. Sample sizes were 127 for Recording vs. Real-Time ° ° Eozs- I =a I : ae : 
Grade 2, 158 for Grade 3, and 162 for Grade 4. ee . . 000" = + = — a CO 


Length x Method 


Passages. Administered via computer: 18 a tg | ai 

passages (3 long, 5 medium, 10 short). - I aE -|—(- tot = 
(1) Short: =25 words, read in entirety. Aci = + ee 
(2) Medium: ~50 words, read in entirety. 


(3) Long: ~85 words, read in entirety. 
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Conclusions 


Scoring. Word accuracy and words correct per ae | 
9 y p > Across grades, significant main effects for passage Error Rate: the proportion of words that were scored |> The timed passage duration was consistently » The Recorded Audio to considered the reference, 


minute (WCPM) were scored by: LENGTH and scoring METHOD, no significant interaction |as incorrect for a given passage. greater (approximately 1-2 secs) the for ASR scoring | because scoring could take place in a quiet setting with 
. ; ; ffect, and mixed results for pairwise comparisons. thods than the Real-Ti i thod no distractions, and the capability to rewind the recording 
1) Real-time: trained human assessors as in . | | metnods than the heal- lime scoring method. : 
hahaa ORE >» Recorded Audio and Real-Time scores were different > The mean error rates ranged from 3% to 10%, > Because the ASR and Recording scoring methods to ensure the most accurate word scores. The primary 
a | ipa oes a deseeiare ors wile sth scores Were |when they were disaggregated by grade, scoring used the same time duration to compute WCPM, this elem ta pec senhpehmasinig Audio scores to 
(2) Audio Recording: trained human assessors Oe ee ea methods, and passage length. would lead to decreased WCPM scores compared to | O°) Ne Meath line ain sama 
: : : > Real-Time scores were higher than both the ASR and . > The agreement rate was quite varied between students 
via audio recordings. Recording > Error rates were highest for ASR. the Real-Time scoring method. for ASR vs. Real-Time, and for ASR vs. Recording 
(3) Automated Speech Recognition (ASR). a » Error rates were lowest for Real-Time scoring. _ | | > The two human scores (Recording and Real-Time) had 
Limitations | | > Error rates were higher for shorter passages. > Although it is not possible to determine the “true” the highest kappa agreement. 
- Same time duration used for ASR and Recordings > Error rates were higher for lower grade levels. passage timed duration, we hypothesize that the > Average ASR vs. Real-Time Cohen’s kappa: Grade 2 = 
Analyses. Mixed model approach with two fst wna ee ee ASR computer-generated time would be the most 82, Grade 3 = .90, and Grade 4 = .91. 
within-subject variables to test the mean WCPM ‘see R 03) will celal Vres scores Uae eee accurate. The ASR time is the duration from the > The ASR may need additional training, especially for 
and error rate differences between passage - Lost scores due to technology benefits comparison of Bielanee OF 2 iis’ Paceage Word: Tene tonminanon apn Lt aa ae A 
' ' 1 1 ‘ : | |- > . 
length, scoring method, and their interaction. The ASR to Recordings (no lost Real-Time data). of the last word read, in centi-seconds. ; hiya elimarshonoe 
length factor included three categories short, considerable resources (time, money, instruction)? 
medium, and long. The scoring method factor 
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